# Distributed Training

Expert Parallelism Load Balancer (EPLB) is a load balancing algorithm for Expert Parallelism (EP) in deep learning. It ensures load balance across different GPUs through a redundant expert strategy and a heuristic packing algorithm, while utilizing group-constrained expert routing to reduce inter-node data traffic. This algorithm is significant for large-scale distributed training, improving resource utilization and training efficiency.

DualPipe is an innovative bidirectional pipeline parallel algorithm developed by the DeepSeek-AI team. By optimizing the overlap of computation and communication, this algorithm significantly reduces pipeline bubbles and improves training efficiency. It performs exceptionally well in large-scale distributed training, especially for deep learning tasks requiring efficient parallelization. DualPipe is developed based on PyTorch, easy to integrate and extend, and suitable for developers and researchers who need high-performance computing.

LLaSA_training is a speech synthesis training project based on LLaMA, aimed at enhancing the efficiency and performance of speech synthesis models by optimizing training and inference computational resources. This project leverages both open-source datasets and proprietary datasets for training, supports various configurations and training methods, and offers high flexibility and scalability. Its main advantages include efficient data processing capabilities, strong speech synthesis effects, and support for multiple languages. This project is suitable for researchers and developers in need of high-performance speech synthesis solutions, applicable to the development of intelligent voice assistants, speech broadcasting systems, and other scenarios.

Memory Layers at Scale is an innovative implementation of memory layers that adds extra parameters to models through a trainable key-value lookup mechanism, without increasing floating-point operations. This method is particularly significant in large-scale language models as it enhances the model's storage and retrieval capabilities while maintaining computational efficiency. The key advantages of this technology include effective model capacity expansion, reduced computational resource consumption, and improved model flexibility and scalability. Developed by the Meta Lingua team, this project is suited for scenarios that handle large datasets and complex models.

PrimeIntellect-ai/prime is a framework designed for efficient, globally distributed training of AI models over the internet. Through technological innovation, it facilitates cross-regional AI model training, improves computing resource utilization, and reduces training costs, which is critical for AI research and application development that requires significant computational resources.

INTELLECT-1-Instruct is a 1 billion parameter language model trained from scratch on 1 trillion English text and code tokens by Prime Intellect. The model supports text generation and has the capability for distributed training, allowing for high-performance training across unreliable, globally distributed workers. It utilizes the DiLoCo algorithm for training and a custom int8 all-reduce kernel to minimize communication load, significantly reducing communication overhead. The background information reveals that it has received computational support from 30 independent community contributors and underwent training across 14 concurrent nodes on three continents.

Meta Lingua is a lightweight and efficient library for training and inference of large language models (LLMs) designed specifically for research purposes. It utilizes easy-to-modify PyTorch components, enabling researchers to experiment with new architectures, loss functions, and datasets. The library aims to facilitate end-to-end training, inference, and evaluation, providing tools for better understanding the speed and stability of the models. Although Meta Lingua is still under development, it already offers several sample applications demonstrating how to use this repository.

Prime Intellect is committed to democratizing AI development on a scalable scale. It offers the discovery of global computing resources, model training, and the capability to co-own smart innovation. By distributing training across clusters, it enables users to train cutting-edge models and co-own the open AI innovation outcomes, including language models and scientific breakthroughs.

OpenDiLoCo is an open-source framework that implements and extends DeepMind’s Distributed Low-Bandwidth (DiLoCo) method, supporting global distributed AI model training. It makes it possible to efficiently train AI models in areas with scattered resources by providing a scalable and decentralized framework, which is significant for promoting the普及 and innovation of AI technology.

Zero Bubble Pipeline Parallelism is a crucial component of large-scale distributed training, and its efficiency is affected by pipeline bubbles. We introduce a scheduling strategy that successfully achieves zero pipeline bubbles under synchronous training semantics. The core idea behind this improvement is to divide backward calculation into two parts: one part calculates the gradients of the input, and the other part calculates the gradients of the parameters. Based on this idea, we manually designed novel pipeline scheduling, which significantly outperforms benchmark methods. We further developed an algorithm that automatically finds the optimal scheduling based on specific model configuration and memory constraints. Furthermore, to truly achieve zero bubbles, we introduce a novel technique that bypasses synchronization during optimizer steps. Experimental evaluation demonstrates that our method achieves up to 23% higher throughput than the 1F1B schedule under similar memory constraints. This number can further increase to 31% when memory constraints are relaxed. We believe our results mark an important step towards realizing the potential of pipeline parallelism.